The Finnish Biodiversity Information Facility as a best-practice model for biodiversity data infrastructures

Biodiversity informatics has advanced rapidly with the maturation of major biodiversity data infrastructures (BDDIs), such as the Global Biodiversity Information Facility sharing unprecedented data volumes. Nevertheless, taxonomic, temporal and spatial data coverage remains unsatisfactory. With an increasing data need, the global BDDIs require continuous inflow from local data mobilisation, and national BDDIs are being developed around the world. The global BDDIs are specialised in certain data types or data life cycle stages which, despite possible merits, renders the BDDI landscape fragmented and complex. That this often is repeated at the national level creates counterproductive redundancy, complicates user services, and frustrates funders. Here, we present the Finnish Biodiversity Information Facility (FinBIF) as a model of an all-inclusive BDDI. It integrates relevant data types and phases of the data life cycle, manages them under one IT architecture, and distributes the data through one service portal under one brand. FinBIF has experienced diverse funder engagement and rapid user uptake. Therefore, we suggest the integrated and inclusive approach be adopted in national BDDI development.peerReviewe

D3.2 DiSSCo Digitisation Guides Website - Consolidating Knowledge on Collections Mobilisation

In order to support the digitisation activities of DiSSCo, we have considered how best to prepare collections for digitisation, digitise them, curate their associated data, publish those data, and measure the outputs of projects and programmes. We have examined options and approaches for different types and sizes of collections, when outsourcing should be considered, and what different project management approaches are most appropriate in this range of circumstances. This report describes the approach we have taken to developing an online community-edited manual, our guidelines, other relevant resources and platforms, and a set of recommendations on how to develop and this work to enhance future digitisation capacity across DiSSCo collectionholding organisations.info:eu-repo/semantics/publishedVersio

Sisällöntuotto-ohje vieraslajiportaaliin

Liite 4. HAVINA-projektin loppuraporttiin URN:NBN:fi-fe2014062529465</a

An Automated Early Warning Biosecurity Alert System for Finland

Effective and timely pest plant and animal control necessitates access to relevant biodiversity data as quickly as possible. For Finland, on Europe's frontier, biosecurity is of particular importance. The Natural Resources Institute Finland and the Finnish Food Authority are the organisations responsible for biosecurity in Finland. The Finnish Biodiversity Information Facility (FinBIF) is a data repository for researchers, government and the public. FinBIF consolidates datasets of living Finland from many providers at a single online source. The data held in FinBIF's data warehouse include the results of national monitoring programs, museum collections and citizen science platforms. On average, over the last five years more than 10,000 occurrence records have been collated by FinBIF per day. These records often include invasive taxa or pest plants and animals that require control. FinBIF acts as bridge between biodiversity data collectors, researchers, other government agencies and the two organisations responsible for Finnish biosecurity. An automated early warning system has been established to alert biosecurity coordinators at the Natural Resources Institute Finland and the Finnish Food Authority as soon an occurrence record of biosecurity concern enters the FinBIF data warehouse. We outline the development of the alert system, its implementation to date, and present some of the lessons learned so far and prospects for the future

Managing Taxon Data in FinBIF

The Finnish Biodiversity Information Facility, FinBIF (https://species.fi), has developed its own taxon database. This allows FinBIF taxon specialists to maintain their own, expert-validated view of Finnish species. The database covers national needs and can be rapidly expanded by our own development team. Furthermore, in the database each taxon is given a globally unique persistent URI identifier (https://www.w3.org/TR/uri-clarification), which refers to the taxon concept, not just to the name. The identifier doesn’t change if the taxon concept doesn’t change. We aim to ensure compatibility with checklists from other countries by linking taxon concepts as Linked Data (https://www.w3.org/wiki/LinkedData) — a work started as a part of the Nordic e-Infrastructure Collaboration (NeIC) DeepDive project (https://neic.no/deepdive). The database is used as a basis for observation/specimen searches, e-Learning and identification tools, and it is browsable by users of the FinBIF portal. The data is accessible to everyone under CC-BY 4.0 license (https://creativecommons.org/licenses/by/4.0) in machine readable formats. The taxon specialists maintain the taxon data using a web application. Currently, there are 60 specialists. All changes made to the data go live every night. The nightly update interval allows the specialists a grace period to make their changes. Allowing the taxon specialists to modify the taxonomy database themselves leads to some challenges. To maintain the integrity of critical data, such as lists of protected species, we have had to limit what the specialists can do. Changes to critical data is carried out by an administrator. The database has special features for linking observations to the taxonomy. These include hidden species aggregates and tools to override how a certain name used in observations is linked to the taxonomy. Misapplied names remain an unresolved problem. The most precise way to record an observation is to use a taxon concept: Most observations are still recorded using plain names, but it is possible for the observer to pick a concept. Also, when data is published in FinBIF from other information systems, the data providers can link their observations to the concepts using the identifiers of concepts. The ability to use taxon concepts as basis of observations means we have to maintain the concepts over time — a task that may become arduous in the future (Fig. 1). As it stands now, the FinBIF taxon data model — including adjacent classes such as publication, person, image, and endangerment assessments — consists of 260 properties. If the data model were stored in a normalized relational database, there would be approximately 56 tables, which could be difficult to maintain. Keeping track of a complete history of data is difficult in relational databases. Alternatively, we could use document storage to store taxon data. However, there are some difficulties associated with document storages: (1) much work is required to implement a system that does small atomic update operations; (2) batch updates modifying multiple documents usually require writing a script; and (3) they are not ideal for doing searches. We use a document storage for observation data, however, because they are well suited for storing large quantities of complex records. In FinBIF, we have decided to use a triplestore for all small datasets, such as taxon data. More specifically, the data is stored according to the RDF specification (https://www.w3.org/RDF). An RDF Schema defines the allowed properties for each class. Our triplestore implementation is an Oracle relational database with two tables (resource and statement), which gives us the ability to do SQL queries and updates. Doing small atomic updates is easy as only a small subset of the triplets can be updated instead of the entire data entity. Maintaining a complete record of history comes without much effort, as it can be done on an individual triplet level. For performance-critical queries, the taxon data is loaded into an Elasticsearch (https://www.elastic.co) search engine

‘As Open as Possible, as Closed as Necessary’ – Managing legal and owner-defined restrictions to openness of biodiversity data

The Finnish Biodiversity Information Facility FinBIF receives, stores and manages biodiversity data mobilised in Finland, and shares the data through its own portal (species.fi) and through Global Biodiversity Information Facility GBIF. FinBIF’s data policy (data policy) embraces the European FAIR data principles (FAIR - Findable, Accessible, Interoperable, Reusable; Wilkinson (2016)) but also incorporates specific restrictions stemming from national legislation, researchers’ needs, and data owners’ requirements. Here, we describe how the necessary, due to various reasons from sensitivity of the data to research embargo, restrictions to openness have been defined and implemented on the policy level and in technical data infrastructure solutions. We hope to contribute to an improvement of data management in the international biodiversity data infrastructures. In Finland, the law prohibits public authorities from distributing occurrence data if this causes increased threat to endangered species. However, neither the definition of ‘endangered species’ nor guidelines for the evaluation of potential risk by openness of data are formulated. To enable mobilisation of datasets containing information on endangered species, FinBIF convened a task force commissioned to set rules on data distribution, which respect the spirit of the law. The task force consisted of representatives of relevant data holding authorities and it consulted a wide group of taxon experts and the species information community. First, a list of species, judged to be among those targeted by the spirit of the law, was created (sensitive species data). Then the rules of restriction were decided on for each of the species. Measures of restriction ranged from complete non-disclosure of data to temporal and spatial restrictions. The identified safeguards concerning the sensitive data management in all use cases led us to create a series of innovative solutions Researchers often wish to restrict the openness of data they have gathered for research purposes These restrictions include embargo periods, limitations on the precision of data and controls on how the data is used. In many cases, however, researchers are willing to allow unrestricted official use of their data in certain cases such as for conservation management or land use planning. In these cases they will often allow storage and restricted use of exact data without an embargo. The same may be true for other data owners, such as nongovernmental organisations (NGOs) or private citizens. To support restrictions to openness, while simultaneously securing mobilisation of valuable datasets, FinBIF applies data sharing contracts including, as a rule, a precondition to share the original data with the public authorities for official use under the Creative Commons 4.0 BY -licence (CC 4.0 BY). The technical solution to enabling the rather complex data policy is that FinBIF stores the collated data in two separate data warehouses: a public one for the distribution of fully open data and temporally and spatially coarsened sensitive data, alongside another containing all data but with restricted access to authorised users. In addition, to allow case-by-case release of restricted data, FinBIF has developed a data request function (Fig. 1). When users of the open data retrieve a dataset using, e.g., taxonomic and spatial filtering, they receive a search result stating whether there are restricted data available based on the filters used. In these cases a user can issue a data request, automatically distributed to all owners of data contained in the collated data batch. Agreeing on the principles about how to apply restrictions to data openness and how to define authoritative use, has not been easy given the lack of precedents. It has required thorough and inclusive consultation with both state administration, conservation practitioners, scientific specialists and lawyers. The two main cultural constraints to overcome have been (1) embracing the FAIR principles of truly ‘as open as possible’ and only ‘as closed as (absolutely) necessary (European Commission 2016); and, perhaps surprisingly, (2) figuring out novel ways to work across different state administrative sectors to share data

Finnish Citizen Science Based Bird Monitoring Schemes and User Interfaces in FinBIF

Monitoring of bird populations is based primarily on volunteer birdwatcher activity in Finland. Hence, development of online user interfaces and data availability have become a priority in order to encourage bird watchers to participate in monitoring schemes. Most Finnish bird monitoring is managed by the Finnish Museum of Natural History LUOMUS, which oversees a wide spectrum of long-running programs including: a bird ringing (banding) scheme running since 1913, a winter bird census established in 1956, a breeding bird census initiated in 1975, a raptor monitoring program started in 1982, and, a nest recording scheme ongoing since the 1940s. In 2018, more than 1,500 volunteer birdwatchers participated in LUOMUS bird monitoring schemes. Data gathered from these programs constitute our basis of knowledge on national bird populations and demographic trends and are actively incorporated in conservation, scientific, land-use planning, and administrative purposes in Finland. In principle, all data are open and freely accessible via the Finnish Biodiversity Information Facility (FinBIF), however, the law prohibits authorities from distributing species occurrence data if this causes an increased threat to certain endangered species. Accordingly, sensitive data details are not available. Reporting valuable fieldwork data can sometimes be demanding. As such, developing user-friendly interfaces for data portals is critical to facilitating volunteer activity. Essential tools for volunteers include a simple login, smooth and augmented data input, automated validation of data, and, perhaps most importantly, ease of access to up-to-date data. Crucial to administrators are system reliability, operability, and easy data management. Comprehensive data validation and visualization tools and extensive search functions aid in revealing errors and thereby increase data quality. Finally, simple query tools and easy access to data are of paramount importance for smooth abd flexible use of the data. Keeping in mind these demands, we have developed the main FinBIF platform and project-specific user interfaces in order to facilitate participation in bird monitoring programs. We will introduce these user interfaces and our achievements and challenges in the development process

A Successful Crowdsourcing Approach for Bird Sound Classification

Automated recorders are increasingly used in remote sensing of wildlife, yet automated methods of processing the audio remains challenging. Identifying animal sounds with machine learning provides a solution, but optimizing the models requires annotated training data. Producing such data can require much manual effort, which could be alleviated by engaging masses to contribute to research and share the workload. Birdwatchers are experts on identifying bird vocalizations and form an ideal focal audience for a citizen science project aiming for the required multitudes of annotated avian audio data. For this purpose, we launched a web portal that was targeted and advertised to Finnish birdwatchers. The users were asked to complete two kinds of tasks: 1) classify if a given bird sound belonged to the focal species and 2) classify all the bird species vocalizing in 10-second audio clips. In less than a year, the portal achieved annotations for 244,300 bird sounds and 5,358 clips, and attracted, on average, 70 visitors on daily basis. More than 200 birdwatchers took part in the classification tasks, of which 17 and 4 most dedicated users produced over half of the sound and clip classifications, respectively. As expected of birder experts, the classifications among users were highly consistent (mean agreement scores between 0.85–0.95, depending on the audio type) and resulted in high- quality training data for parameterizing machine learning models. Feedback about the web portal suggested that additional functionality such as increased freedom of choice would increase user motivation and dedication.Peer reviewe

Notebook: Customizable web forms for recording observations

’Notebook’ is one of the primary data management systems of the Finnish Biodiversity Information Facility (FinBIF). It is a web solution for recording opportunistic as well as sampling-event-based species observations. It is being used for systematic monitoring schemes, various citizen science projects, and platforms for species enthusiasts. Notebook's main software component is LajiForm, which is the engine that renders a given JSON Schema into a web form. LajiForm is a separate, reusable module that is fully independent from other FinBIF systems. Notebook as a whole, includes other features embedded in FinBIF, such as linking users' geographical data to observation documents, spreadsheet document importing and form templates. We will demonstrate how the Notebook system works as a whole and also focus on LajiForm's technical aspects (Fig. 1). All Notebook forms use FinBIF's ontological schema in JSON Schema format. Rendering user-friendly web forms based on a single schema is a difficult task, because the web form should be asking meaningful questions, instead of just rendering the schema fields according to the form description. We want to present questions in an interactive style. For instance, after drawing a geographical location on a map for a potential flying squirrel nesting tree, we would ask "did you see droppings at the nest?", and answering "yes" would update the document to include a flying squirrel taxon identification with fields "breeding" and "record basis" filled in but not rendered to the form. A simpler form engine without a user interface (UI) customization layer would just render the "taxon", "breeding" and "record basis" fields and the user would have no understanding why there are so many fields to answer and how they relate to their work or study. Some forms are complex, e.g., for experienced biology enthusiasts who need a form that is advanced, customizable, and compact. Some forms are simple, e.g., for elementary school children. To tackle these challenges, LajiForm uses a separate schema for UI that allows everything from simple customization like: defining widgets for fields (e.g., date widgets, taxon autocomplete widget, map widget), changing field order or customizing field labels; to more complex customization like transforming the schema object structure, defining conditions when certain fields are shown or if updating a field should have an effect on other fields. All the functionality is split into a loosely coupled collection of components, which can be either used as standalone components or composed together in order to achieve more advanced customization. The programming philosophy has drawn inspiration from functional programming, which has been helpful in writing isolated, composable functionality. LajiForm is written with the JavaScript framework React. LajiForm is built on top of react-jsonschema-form (RJSF), which is an open source JSON schema web form library provided by Mozilla. RJSF handles only simple customization, but it is very flexible in design and allows us to build extensions with features that are more powerful. Some features and design proposals were submitted to Mozilla – FinBIF is the largest code contributor to RJSF outside of Mozilla, with a dozen pull requests merged

Species Threat Assessment Tool and Online Result Service in FinBIF

The threat assessment of Finnish species was carried out during 2017–2018 for the third time using International Union for Conservation of Nature (IUCN) criteria (IUCN 2012a, IUCN 2012b, IUCN 2016). The Red List was published in March 2019 (Hyvärinen et al. 2019). In conjunction with the assessment of threatened species, 36,602 species or lower taxa were listed. The information was sufficient for assessment of 22,418 species. It was done by 18 expert groups of different organism groups. The process was coordinated by the Finnish Environment Institute Syke and led by the steering group of the assessment from the Ministry of Environment. For the first time, the Finnish Biodiversity Information Facility FinBIF offered a documentation tool and archive for the threat assessment. The assessment was based on the national checklist of Finnish species coordinated by FinBIF. Many of the expert groups are in active collaboration with FinBIF in maintaining FinBIF’s taxon database and in updating the checklists. Hence, there was a good foundation to build on in developing the cooperation further and deepening the integration of the Red Listing process into FinBIF’s IT infrastructure. The documentation tool of the assessment is implemented in the taxon database of FinBIF (Fig. 1). The Red List data of the 2010 (all species) and 2015 (birds and mammals) assessments are readily available in the tool. The assessor can therefore easily copy and confirm e.g. area of occupancy, extent of occurrence, generation length and habitat of a species, if there is no need for editing. The service offers the possibility to add notes to most of the fields separately and commenting on the assessment by other authorized users. The tool archives the history of all changes. In line with the IUCN instructions, the tool automatically chooses the criteria leading to the highest possible threat category of criteria A-E filled out for each species. However, the assessor confirms the final evaluation. Finally, in several fields, the tool automatically checks the validity of values entered, e.g. criteria, threat category, length of the observation period, causes of threat, and current threat factors. The tool includes necessary fields for back-casting the categories of previous assessments to count the Red List Index. There is also a possibility to add or choose references for the assessment of a certain species in the publications part of the taxon database. Due to linkage through the taxon database, the updated threat categories of each Finnish species are immediately available as additional information of each species introduced in FinBIF. Also for the first time, the results of the threat assessment can be examined online directly after its publication at the Red List online service through FinBIF: https://punainenkirja.laji.fi/en. The online service makes Red List categories and related criteria searchable. Data can be categorized also by habitat, causes of threat, or current threat factors. Due to the ability to conduct searches, the online service supplements the printed book (Hyvärinen et al. 2019), which includes extensive summaries for groups of organisms